Background

Segmenting continuous streams

Speech and action sequences are both continuous information streams that must be successfully segmented into constituent sub-units in order to be understood…

Unsuccessful speech segmentation.

Successful action segmentation.

In both the speech and action domain, we know this segmentation task is achieved via a combination of top-down and bottom-up processing.

Top-down and bottom-up

Top-down processing involves the application of pre-existing knowledge to determine where boundaries between phrases occur. Bottom-up processing involved properties of the stimulus to determine boundary location.

Work with adults has highlighted top-down and bottom-up cues that support segmentation of speech and action. For example:


Speech	Listeners apply their knoweldge of word meanings and grammar to determine the locations of boundaries in speech (e.g. Mattys et al., 2007).	Prosodic cues (e.g. pasue and pre-boundary lengthening) are produced at phrase boundaries (e.g. Wagner & Watson, 2010), and listeners detect these cues to determine the location of phrase boundaries in speech (e.g. Schafer et al., 2000).
Action Sequence	Observers track the goals and intentions of an actor, and map boundaries to moments of goal-achievement (e.g. Levine et al., 2017).	The movement itself contains kinematic cues to boundaries between actions. These kinematic cues include pause and pre-boundary lengthening (Hilton et al., 2019), and observers make use of these kinematic cues to determine the location of boundaries in the action sequence (Hemeren & Thill, 2010).

Developmental Perspective

Infants’ access to top-down processes is restricted, because they do not yet possess the knowledge/experience.
In speech, it has been proposed that infants therefore initially capitalise on bottom-up cues (prosody) to segment the stream (Prosodic Bootstrapping Account; Gleitman & Wanner, 1982 )
However, little is known about infants’ processing of bottom-up cues during action segmentation.
In parallel to early speech processing, infants may capitalize on kinematic boundary cues to initially segment actions, especially when the actions are unfamiliar or not goal-directed.

Research Questions and Aims

Are infants sensitive to kinematic boundary cues?
- Measuring the electrophysiological response to kinematic boundary cues.
- Examining whether the response is similar to that evoked by prosodic boundary cues.
Do the kinematic boundary cues modulate processing of the subsequent action?
- Finding that infants are sensitive to kinematic boundary cues would not automatically mean that these cues play a role in action processing.
- We therefore examined whether the kinematic boundary cues modulate action processing, by examining electropsyshiological response to actions that do/do not follow kinematic boundary cues.

Procedure

Row

Stimuli

Three child friendly characters were created:

character 1

character 2

character 3

These characters were then animated to perform sequences of three actions.
Two action sequences were defined:

Turn then stretch then jump
Jump then stretch then turn

On no-boundary trials each sequence was shown as a single continuous sequence.
On boundary trials, a boundary was signalled between the second and final action.
On boundary trials, the boundary was signalled by two kinematic boundary cues:

Pre-boundary lengthening: To achieve pre-boundary lengthening, the second “half” of the pre-boundary action (stretch) was extended by 240 ms, by slowing down the rate at which the character returned to its normal size.
Pause: Following the completion of the pre-boundary action, the character paused motionless for 350 ms.

These timings were based on typical durations of pre-boundary lengthening and pause as found in naturally-produced speech, and durations of the actions forming the sequences were:

	No-boundary Trial	Boundary trial
Element	Duration (ms)	Duration (ms)
still frame	1000	1000
action 1	600	600
action 2	600	840
pause	0	350
action 3	750	750

Each character performed both sequences with and without a boundary, resulting in a final stimuli set of 12 videos (3 characters x 2 sequences x 2 trial types).
Below, you can see an example of a no-boundary trial and a boundary trial.
- Note: On web-browsers, these videos do not play at their full time-resolution, meaning that they may appear somewhat jumpy. When presented to participants in the lab, the videos were correctly rendered and smooth.

Row

No-boundary trial

Boundary trial

Row

Participants

12-month-old infants (N = 23) from German-speaking households contributed data.

Summary statistics: Number of artefact-free trials contributed by participants
Condition	Mean no. of trials	Range
no-boundary	23.2	15 - 34
boundary	25.0	14 - 37

EEG testing

Infants were shown the 12 stimulus videos in a ranomized order until the infants became bored and thus looked consistently away from the screen. Including breaks and pauses, we typically were able to record EEG from the infants for ~ 10 minutes.
EEG was recorded from 30 electrodes.
9 of these electrodes served as critical electrodes for analysis (F3, Fz, F4, C3, Cz, C4, P3, Pz, P4).

Question 1

Were infants sensitive to the kinematic boundary cues?

To answer this question, we examined the ERP from the onset of the pre-boundary lengthening (i.e. the mid-point of the second action) until the offset of the final action.
To examine whether infants processed the kinematic boundary cues, we examined the ERP in this time interval for evidence of the Closure Positive Shift.

Closure Positive Shift (CPS):

An ERP component initially discovered in response to prosodic boundary cues in speech (Steinhauer et al., 1999) . This component is a slow, broadly distributed positivity in the ERP that begins around the onset of the boundary and lasts approximately 500 ms (Boegels et al., 2011) .

Recent work has extended our understanding of the CPS in two critical ways:

The CPS has been found in response to prosodic boundary cues in speech already during the first year of life (Holzgrefe et al., 2018).
In adults, a CPS-like positivity has been found in response to kinematic boundary cues in action sequences (Hilton et al., 2019).

We wanted to bring these recent advances together, by testing whether a boundary in an action sequence evokes a CPS-like positivity already during infancy

Column

Slow-motion video - ERP for whole sequence

Column

ERP - Regions

ERP - Individual Electrodes

Column

Analysis & Conclusion

For every trial, segments between the mid-point of the second action and the mid-point of the third action were exported for analysis. The maximum amplitude in each segment was calculated, resulting in an analysis of mean maximum amplitude during this time interval.

Repeated measures ANOVA (condition: no-boundary vs. boundary x region: frontal vs. central vs. posterior):

effect	F	df	p	\(\eta_{G}^{2}\)
condition	73.11	1, 26	<.001	0.342
region	19.08	2, 52	<.001	0.065
condition*location	2.70	2, 52	0.077	0.004

These results indicate a positive shift in the boundary condition relative to the no-boundary condition.
This difference in the ERPs indicates that the infants detected the kinematic boundary cues.
The ERP response to kinematic boundary cues is CPS-like, suggesting that the cognitive processes underlying the processing of kinematic cues are similiar to those involved in the processing of prosodic boundary cues.

Question 2

Do kinematic boundary cues modulate processing of subsequent actions?

The finding that infants are sensitive to kinematic boundary cues does not neccessarily mean that these cues are involved in action processing
We therefore wanted to test whether kinematic boundary cues affect processing of the actions that form the sequence.
We compared response to all three actions in the sequence.
The kinekmatic boundary cues do not occur until the offset of the second action.
If kinematic boundary cues affect action processing, we should therefore find an effect of condition only on actions following the boundary cues (i.e. the final action).
We took the negative central component as a candidate component by which to examine infants’ encoding of the individual actions

Negative central (Nc) component:

A negative peak in the ERP over fronto-central electrodes emerging between 300 and 900 ms following stimulus onset, implicated in attentional processing (e.g., Nelson & Collins, 1991; Reynolds & Richards, 2005) . Has recently been taken as a measure of action processing during infancy, reflecting attention to and encoding of an individual actions (Monroy et al., 2019).

Visual inspection of the ERPs indicates that the individual actions prior to the kinematic boundary cues evoke an Nc component.

Column

ERP

Column

Analysis & Conclusion

For every trial, the Nc was analysed by exporting the minimum amplitude from the ERP in the 250 ms - 750 ms time interval following the onset of each action. The mean minimum amplitude was then averaged across six fronto-central electrodes (F3, Fz, F4, C3, Cz, C4), and analysed with a 3 (action: first, second, final) x 2 (condition: boundary, no-boundary) repeated measures ANOVA.

effect	F	df	p	\(\eta_{G}^{2}\)
condition	0.65	1, 26	0.426	0.003
action	4.16	2, 52	0.021	0.021
condition*action	7.73	2, 52	0.001	0.026

These results indiacte that each action evoked an Nc-component, except the final action in the no-boundary condition.
The kinemtic boundary cues modulated processing of the subsequent action.
The final action in the no-boundary could have been encoded as a continuation of the second action, hence no Nc-response.
Alternatively, the final action in the no-boundary condition may have overloaded infants’ processing capacity. Kinematic boundary cues in the boundary condition could however have propmpted the chunking and storage of previous actions, freeing-up capacity for the final action.

tl;dr

We are interested in how infants process boundaries between individual actions of an action sequence.

Work with adults suggests that kinematic cues (properties of the movement) can signal the location of boundaries in action sequences.

We presented 12-month-old infants with cartoon action sequences while recording EEG.

Half of the sequences contained kinematic boundary cues (pre-boundary lengthening and pause).

We found evidence of an ERP component indicating the infants detected and processes the kinematic boundary cues.

The kinematic cues also modulated infants’ processing of subsequent actions.

We contend that these low-level kinematic cues play a role in early action segmentation and processing.

Poster

Link to poster download

References from poster

Twitter Icon on pdf from Freepik from www.flaticon.com